Cognitive state classification in a spoken tutorial dialogue system

This paper addresses the manual and automatic labeling, from spontaneous speech, of a particular type of user affect that we call the cognitive state in a tutorial dialogue system with students of primary and early middle school ages. Our definition of the cognitive state is based on analysis of children's spontaneous speech, which is acquired during Wizard-of-Oz simulations of an intelligent math and physics tutor. The cognitive states of children are categorized into three classes: confidence, puzzlement, and hesitation. The manual labelling of cognitive states had an inter-transcriber agreement of kappa score 0.93, which was higher than strong emotion labelling in literature. For the automatic labelling, text generated by an automatic speech recognizer is searched for keyword classes and part-of-speech sequences; speech signal itself is analyzed in order to identify cepstral and prosodic correlates of cognitive states. Our study also proposes a set of cepstral features based on cognitive state-dependent speech recognition, in which the phoneme models are adapted to utterances categorized into the corresponding cognitive states. The effectiveness of the proposed method has been tested on both manually and automatically transcribed speech, and the test yielded very high correctness: 96.6% for manually transcribed speech and 95.7% for automatically recognized speech. Our study shows that the proposed cepstral features greatly outperformed the other types of features in the efficiency of cognitive state classification. Our study also shows that spectral and prosodic features derived directly from speech signals were very robust to speech recognition errors, much more than the lexical and part-of-speech based features.